Image capturing and display apparatus and method

ABSTRACT

Provided is a display apparatus and method. The display apparatus may sense light reflected from an object and passed through a display panel, and may control a power of a backlight unit depending on whether the light has passed through the display panel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of Korean PatentApplication No. 10-2010-0049564, filed on May 27, 2010, in the KoreanIntellectual Property Office, the disclosure of which is incorporatedherein by reference.

BACKGROUND

1. Field

One or more embodiments relate to a display apparatus and method, andmore particularly to a display apparatus and method simultaneously senseincident light to generate an image and displaying an image.

2. Description of the Related Art

A display apparatus may include a display panel, a camera, and abacklight unit. The display apparatus may operate in a capturing modefor acquiring image data, or in a displaying mode for displaying theimage data.

Two examples of such display apparatus have been discussed in an articleby Hirsch et al., “A Thin, Depth-Sensing LCD for 3D Interaction usingLight Fields,” in Proceedings of SIGGRAPH ASIA December 2009, a similardisplay apparatus was set forth. In a first example, the displayapparatus sequentially includes an angle limiter layer, a displaypanel/a coded mask layer, and a diffusing layer. The display panel iscontrolled to generate a coded mask effect of the coded mask layer, anda camera would be positioned behind the diffusing layer. The capturedimage data would thus be obtained from the surface of the diffusinglayer. In this approach, it was necessary to position the camera a meterbehind the diffusing layer to compensate for a lack of an angle-limitingfilm, which the authors of the article indicated was necessary, ratherthan suffering the downside affects of using an angle-limiting film infront of the display.

In an alternative approach, the display device sequentially includes thedisplay panel, a mask, a single large-format sensor, and a backlight.The single large-format sensor detects light that is incident on thesensor after passing through the display panel. The authors indicatedthat it was necessary to have the single large-format sensor be made upof a plane of clear photo-detectors in a clear substrate or membrane, asit was necessary that the most light from the backlight be provided tothe display panel. In these two approaches, the external camera and thesingle large-format sensor both resulted in a low resolution capturingcapabilities, which was not as much of a concern to the authors becausethey were more focused on obtaining sufficient orthographic images forgesture detection, which they acknowledged did not require higherquality images.

The Hirsch et al. article additionally notes that the utilization of adense camera array behind the display panel may not be practical and maybe difficult to develop, as the approach would increase the engineeringcomplexity compared to the use of the single large-frame sensor. Asfurther noted by Hirsch et al., such a different sensor approach wouldintroduce backlight non-uniformity issues due to sensor element blockinglight generated by the backlight. Rather, the authors of Hirsch et al.pointed out the benefits of the transparent photodiode singlelarge-format sensor, with a clear substrate or membrane, wasspecifically used so light from the backlight would not be blocked. Theauthors of Hirsch et al. have also noted that alternative technologiesmay not be available in the near future. The authors further note thattheir approach requires an extended image data capturing period when amodified uniformly redundant array (MURA) coding pattern is used, withthe entire display panel being set to produce the MURA coding patternfor a few milliseconds, producing noticeable flickering. The extendedperiod of time required for the MURA coding patter is required so thesingle large-frame sensor can uniformly capture image data that has beenmodulated by the tiled-MURA code.

Accordingly, a display apparatus including a camera or such a singlelarge-frame sensor may have several limitations in acquiring an image,including not being able to accurately sense incident light while alsooperating the display panel to display images based on light radiatingfrom the backlight.

SUMMARY

According to an aspect of one or more embodiments, there may be provideda display apparatus, including a sensor panel configured to sense afirst incident light, incident to the sensor panel subsequent to passingthrough a display, and to at least partially block transmission of lightthrough the sensor panel, and a controller to selectively control thedisplay to be configured to pass the first incident light toward thesensor panel and to control the display to be configured to pass lightout of the display to display an image, wherein the sensor panelincludes plural distinct portions configured to each forward light fromthe sensor panel towards the display.

According to an aspect of one or more embodiments, there may be provideda display apparatus, including a display panel, a sensor panel includingplural sensor units configured to sense incident light from the displaypanel and plural distinct portions configured to each forward light fromthe sensor panel towards the display panel, and a controller toselectively control opacities of pixels and/or sub-pixels of the displaypanel to produce patterns on the display panel for displaying an imageon the display panel, in a second mode, and to form at least one patternon the display panel controlled to be transparent and an area of thedisplay panel surrounding the at least one pattern controlled to beopaque, to capture an image by the at least one sensor unit, in a firstmode, wherein the plural distinct portions are respectively pluralapertures or plural backlight units, with one or more of the pluralapertures including a diffuser element, and with each of the pluralbacklight units respectively generating and directing light toward thedisplay panel.

According to an aspect of one or more embodiments, there may be provideda display method, including selectively configuring a display to form apattern with at least one transparent pixel or sub-pixel to pass a firstincident light through the display and toward a sensor panel, behind thedisplay, and to form an opaque area that at least surrounds the pattern,when a current mode is a first mode, and configuring the display to passlight from a backlight, behind the display, through the display, whenthe current mode is a second mode, sensing the first incident light,having passed through the display, upon incidence to the sensor panelthat at least partially blocks transmission of light, in a first mode,and displaying an image on the display by passing light in a directionfrom the sensor layer through the display, and repeatedly controllingthe current mode to only one of the first mode and the second mode,including at least a changing of the current mode plural times.

The method may include controlling, at a current time in the first mode,the display to have at least one pattern different than a pattern formedat a previous time when the current mode was the first mode, by changingat least one of a number of first pixels of the display that areselectively controlled to be opaque during the first mode, a number ofsecond pixels of the display that are selectively controlled to betransparent during the first mode, positions of one or more pixels ofthe display that are controlled to be opaque during the first mode, andpositions of one or more pixels of the display that are controlled to betransparent pixels during the first mode, than in the previous time.

The method may further include controlling, at a current time in thefirst mode, the display to have at least one pattern different than apattern formed at a previous time when the current mode was the firstmode, by changing at least one of a number of first sub-pixels of thedisplay that are selectively controlled to be opaque during the firstmode, a number of second sub-pixels of the display that are selectivelycontrolled to be transparent during the first mode, positions of one ormore sub-pixels of the display that are controlled to be opaque duringthe first mode, and positions of one or more sub-pixels of the displaythat are controlled to be transparent pixels during the first mode, thanin the previous time.

The method may further include selectively controlling a combination ofa capture rate, for capturing images from the sensor panel, and adisplay rate, for displaying plural images, to be greater than a maximumrefresh rate of the display.

The method may still further include selectively controlling the displayto only pass the first incident light toward the sensor panel in a firstcycle, of plural fixed time length cycles, and controlling the displayto only pass the light out of the display to display the image in asecond cycle, of the plural fixed time length cycles, wherein the fixedtime is a time it takes for the display to perform one refresh. Themethod may include selectively controlling opacities of pixels and/orsub-pixels of the display to produce patterns on the display fordisplaying the image on the display, in a second mode, and to form atleast one pattern on the display controlled to be transparent and anarea of the display surrounding the at least one pattern controlled tobe opaque, to capture an image by at least one sensor unit of thesensor, in a first mode, wherein the display is controlled differentlyin the first mode and the second mode.

Additionally, the method may include determining a depth of an objectrepresented in the first incident light based upon at least one detectedsize of a blur of the at least one pattern detected by the sensor panel,and the determining may further include determining plural depthmeasurements of the object represented in plural first incident lightseach sensed by the sensor panel based respectively upon at least onedetected size of a blur pattern for each sensed incident light, andgenerating 3D information of the object. Still further, the method mayinclude monitoring a movement by the object over plural captured imagesand respectively generated plural 3D information and determining whetherthe monitored movement and generated plural 3D information indicatesthat a gesture, of plural defined gestures, is being made by themovement of the object, and controlling an operation of a device basedon the determined gesture.

The method may include restoring the captured image by calculating adistance-based impulse response based on the determined depth,performing devolution of the captured image using the distance-basedimpulse response to compensate for refraction and/or geometric effectscaused by the at least one pattern of the display, and encoding therestored captured image.

According to an aspect of one or more embodiments, there may be provideda display method, including selectively configuring a display to form apattern with at least one transparent pixel or sub-pixel to pass a firstincident light through the display and toward a sensor panel, behind thedisplay, and to form an opaque area that at least surrounds the pattern,when a current mode is a first mode, and configuring the display to passlight from plural backlights of the sensor panel through the display,when the current mode is a second mode, sensing the first incidentlight, having passed through the display, upon incidence to the sensorpanel that blocks transmission of light, in a first mode, and displayingan image on the display by passing respectively generated light fromeach backlight from the sensor layer through the display, and repeatedlycontrolling the current mode to be one of the first mode and the secondmode, including at least a changing of the current mode plural times.

Additional aspects and/or advantages of one or more embodiments will beset forth in part in the description which follows and, in part, will beapparent from the description, or may be learned by practice of one ormore embodiments of disclosure. One or more embodiments are inclusive ofsuch additional aspects.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages will become apparent and morereadily appreciated from the following description of embodiments, takenin conjunction with the accompanying drawings of which:

FIGS. 1A and 1B illustrate a display apparatus and method, according toone or more embodiments;

FIGS. 2A-2B, 3A-3B, 4A-4B, and 5A-5B illustrate one or more sensorpanels, such as the sensor panel 107 of FIG. 1A, included in a displayapparatus, according to one or more embodiments;

FIGS. 6A through 6C and FIG. 7 illustrate patterns formed by a displaypanel, such as the display panel 103 of FIG. 1A, in a display apparatus,according to one or more embodiments;

FIGS. 8 through 10 illustrate patterns formed by a display panel, suchas the display panel 103 of FIG. 1A, in a display apparatus, accordingto one or more embodiments; and

FIGS. 11A through 11C illustrate a display device, system, and method,according to one or more embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to one or more embodiments,illustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. In this regard, embodimentsof the present invention may be embodied in many different forms andshould not be construed as being limited to embodiments set forthherein. Accordingly, embodiments are merely described below, byreferring to the figures, to explain aspects of the present invention.

FIGS. 1A and 1B illustrate a display apparatus and one or morecorresponding image capturing and/or displaying methods, according toone or more embodiments.

Referring to FIG. 1A, the display apparatus 101 may include a displaypanel 103, a sensor panel 107, a backlight unit 113, and a controller115, for example. In an embodiment, the sensor panel may include one ormore sensor units 109 and one or more apertures 111. The leftillustration of FIG. 1A shows a more expanded representation of thedisplay apparatus 101, while the right illustration shows how layers ofthe display apparatus can be manufactured. In an embodiment, themanufacturing of the right illustration of FIG. 1A, of the display panel103, sensor panel 107, and backlight unit 113, may be performed by anyconventional semiconductor or optical/display fabrication methodology.According to one or more embodiments, the below differing arrangementsof the display panel 103, sensor panel 107, sensor units 109, apertures111, backlight units 113, and any potential diffusing layers therein,may equally be manufactured in the inventive arrangements by suchsemiconductor or optical/display panel fabrication methodologies. Asshown in the right illustration of FIG. 1A, in an embodiment, the sensorpanel may be constructed of a substrate that can be etched orphoto-resist washed to form the apertures 111, so there may be fixedareas where light may be transmitted through, e.g., from the lowerbacklight unit 113 toward to display panel 103, and fixed areas wherelight would be blocked. Additional, as only an example, if a diffuserwere incorporated into one or more apertures, this incorporation couldalso be performed at the time the sensor panel 107 is manufactured, suchthat when light passed through the one or more apertures the light wouldbe diffused by the diffuser.

In one or more embodiments, the display panel 103 may be a liquidcrystal display (LCD) panel, though embodiments are not limited to thesame. In an embodiment, the display panel 103 may be any panel that mayselectively provide one or more pinholes and/or masks enabling thepassing of incident light. Here, the display panel 103 may operate in acapturing mode or in a displaying mode based on a predeterminedcriterion of the controller 115. For example, when the display panel 103refreshes at a frequency of 120 Hz, the display panel 103 may switchbetween the capturing mode and the displaying mode every refresh overthe total 120 Hz, e.g., totaling 60 Hz in the capturing mode and 60 Hzin the displaying mode. In an embodiment, the refresh rate of thedisplay panel 103 is the number of times per second in which the displaypanel 103 can display the data it is given. When the display panel 103is an LCD panel, for example, the refresh rate may be the number oftimes the LCD panel can change states per second, e.g., changing a pixelor sub-pixel from an opaque state to a transparent state, or anyavailable gradation there between, which may be representative ofcombined times it takes all underlying control operations to beperformed.

Accordingly, the pinholes and/or masks may be selectively provided bythe display panel 103 without all or some of the pinholes or masks inone cycle, e.g., while radiating light out of the display 103, and thepinholes and/or masks may be selectively provided by the display panelwith all or some of the pinholes or masks in another cycle, e.g.,selectively passing light through the display panel 103 toward thesensor panel 107. It may not be required that the pinholes or masks arecompletely removed during the radiating cycle, and it is not necessarythat all available, or the same, pinholes or masks be maintained inevery capturing cycle. Alternatively, in an embodiment, all pinholes ormasks are removed in the displaying mode.

In the capturing mode, the display panel 103 may pass an incident firstlight, e.g., as reflected from an object, to the sensor panel 107.Specifically, the display panel 103 may be controlled to form one ormore patterns for passing the incident first light by controlling afirst pixel or a first sub-pixel, for example, to be opaque and bycontrolling a second pixel or a second sub-pixel, also as an example,different from the first pixel, to be transparent. For example, thedisplay panel 103 may be controlled to form a pattern 105, including acircular hole pattern, and may pass the incident first light through thepattern 105. Here, the pattern 105 may be formed by the second pixel,and though the second pixel has been described as being only a singlepixel or sub-pixel, one or more embodiments include plural neighboringsecond pixels, e.g., which could be used to control the aperturediameter of the pinhole represented by the pattern 105. Likewise,depending on embodiment and differing uses of the ultimately sensedlight, the pattern 105 would not be limited to having only a circularform, and may be non-circular. In an embodiment, the size of the pattern105 may be controlled to have a same size or number of pixels/sub-pixelsfor different sized display panels and may be selectively modified basedon differing resolutions of each display to maintain the same patternsize.

Accordingly, in one or more embodiments, the display panel 103 may becontrolled to form various types of patterns. As only examples, thedisplay panel 103 may be controlled to form the pattern including atleast one of a circular hole, a polygonal hole, and a modified uniformlyredundant array (MURA) by combining the first pixel and the secondpixel. Here, the pattern 105 indicates an optical meaning of a patternthat may pass light instead of blocking light.

The display panel 103 may be controlled to repeatedly configure theformed pattern based on a number of incident first lights to be passed.For example, when two incident first lights are desired to be passed,the display panel 103 may be controlled to provide two patterns. Whenfour incident first lights are desired to be passed, the display panel103 may be controlled to provide four patterns.

The display panel 103 may be controlled to form various types ofpatterns by changing at least one of a number of opaque first pixels, anumber of transparent second pixels, a position of each opaque firstpixel, and a position of each transparent second pixel.

The display panel 103 may be controlled to form a plurality of patterns,and may also change a number of patterns or a position of each patternat predetermined intervals. For example, the display panel 103 may becontrolled to form the plurality of patterns horizontally or vertically,or in any other organized arrangement. In one or more embodiments, someor all of the patterns or corresponding positions may be randomly choseby the controller 115, and/or some of the patterns or correspondingpositions may vary between capturing cycles.

In the displaying mode, the display panel 103 may be controlled toclearly outwardly display image data using a second light emitted fromthe backlight unit 113, radiating light outward toward the display panel103.

The sensor panel 107 may be disposed after or below the display panel103 to acquire image data by sensing the incident first light, e.g.,reflected from the object, and passed through the display panel 103toward the sensor panel 107.

The sensor panel 107 may include one or more sensor units 109 to senseincident light and apertures 111 to pass light radiating from thebacklight, for example, radiating toward the display panel 103. In oneor more embodiments, the sensor panel 107 may be configured such thatthe sensor units 109 and the apertures 111 form a grid pattern, or maybe configured so the sensor units 109 and the apertures 111 form arepeating pattern, noting that alternative patterns are also available.In one or more embodiments, the placement of the sensor units 109 maynot be evenly distributed over the entire sensor panel 107, and may beof increased density in one or more areas compared to the density of theapertures 111 in the one or more areas. Further, in an embodiment,portions of the sensor panel 107 may be made up of all or primarilyapertures 111 or sensor units 109. For example, in an area that maycorrespond to the level of a user's eyes, there may be an increaseddensity of the sensor units 109, so a remote recipient would have thefeeling that the user is looking directly at the camera, and directly atthe recipient. Alternative reasons for changing the densities areequally available.

The sensor units 109 may be represented by at least one sensor, whichmay sense the incident first light, e.g., reflected from an object, aspassed through the display panel 103. As described above, one or moresensor units 109 may be arranged to sense the respective incident firstlight passed through the pattern 105 formed in the display panel 103.Here, when there exists a plurality of patterns 105, such as pluralholes, one or more sensor units 109 may sense the respective incidentfirst lights passed through the plurality of patterns. The number ofsensor units 109 that may be needed to sense the incident light from onesuch pattern may be calculated, or estimated, and the sensor panel 107configured accordingly. In one or more embodiments, the sensor layer 105may be reconfigurable and not fixed as more or less sensor units 109 andmore or less apertures 111 may be selectively used or available.

When the pattern 105 of the pattern is formed based on a pixel unitincluding red (R), green (G), and blue (B), the sensor unit 109 mayacquire image data corresponding to R, G, and B by sensing the incidentlight that has passed through the pattern 105. When the pattern 105 isformed based on a sub-pixel unit, a pattern formed as an R sub-pixel, apattern formed as a G sub-pixel, and a pattern formed as a B sub-pixelmay be sequentially formed in different cycles, for example, or throughneighboring sub-pixels of different colors. Here, to increase luminanceof the captured image, the number of times incident light from the Gsub-pixel is captured may be greater than the respective number of timesincident light from the R or B sub-pixels, within the example 120available cycles. Accordingly, the sensor unit(s) 109 may acquire imagedata corresponding to R, G, and B by sensing the incident light that haspassed through the patterns of the display panel 103.

In an embodiment, such as when the display panel 103 does not include oris controlled to not apply respective color filters to the differentpixels or sub-pixels, each sensor unit 109 may be configured to includea particular color filter to acquire image data corresponding to acorresponding color of the color filter. For example, one sensor unit109 may include a first sensor unit and a second sensor unit. Each ofthe first sensor unit and the second sensor unit may include adifferently colored unit color filter. Here, there may further be athird sensor unit with still another differently colored unit colorfilter. Also, each of the first sensor unit and the second sensor unitmay include a plurality of color filters having different colors.Additionally, as noted above, the densities and arrangements of thedifferently colored filters and corresponding sensor units 109 is notrequired to be equal or even, and their densities and arrangements mayvary based upon embodiment.

When the sensor unit 109 is configured to have an R color filter, thesensor unit 109 may acquire image data corresponding to the color red bysensing the incident light that has passed through the pattern 105. Whenthe sensor unit 109 is configured to have a B color filter, the sensorunit 109 may acquire image data corresponding to the color green bysensing the incident light that has passed through the pattern 105. Whenthe sensor unit 109 is configured to have a B color filter, the sensorunit 109 may acquire image data corresponding to the color blue bysensing the incident light that has passed through the pattern 105. Inone or more embodiments, the arrangement of different filtering sensorunits 109 on the sensor panel 107 may be according to different Bayerpatterns, for example.

When the backlight unit 113 is disposed behind or below the sensor panel107, the aperture 111 of the sensor panel 107 may pass the radiatingsecond light emitted from the backlight unit 113 towards the displaypanel 103. Here, the aperture 111 may be represented by an opticalmember that may selectively pass light, always pass light, or merelyrepresented by an emptiness in the sensor panel 107, instead of blockingthe light. The optical member may have one or more lens features fordirecting light from the backlight unit 113. The aperture 111 may besimilar to the pattern 105 of the display panel 103, and in one or moreembodiments may be controllable to be transparent for the displayingmode and blocking of light in the capturing mode. In an alternateembodiment, the aperture is a fixed optical member that always permitslight to be passed.

Accordingly, when the backlight unit 113 is disposed behind or below thesensor panel 107, the aperture 111 enables the second light emitted fromthe backlight unit 113 and radiating towards the display panel 103 to betransferred to the display panel 103 without being blocked by the sensorpanel 107. Accordingly, the display panel 103 may clearly display theimage data through the apertures 111 even when the sensor units 109block transmission of one or both of the first incident light and thesecond light. Similar to above, the densities of sensing units 109compared to the densities of the apertures 111 may be different, andtheir arrangements are not required to be fixed, equal, or even. In anembodiment, as only an example, the sensor panel is configured toinclude or use a greater number of apertures 111 than the number ofsensor units 109 so as to increase display resolution, or configured toinclude or use a greater number of sensor units 109 than apertures 111,so as to increase captured image resolution.

According to another embodiment, the sensor panel 107 may furtherinclude, in the aperture 111, a diffuser to prevent the radiating lightfrom the backlight unit 113 from being partially concentrated, or thesensor panel 107 may be configured to include the backlight unit 113,e.g., in the position of the aperture 111, instead of using the aperture111.

The display apparatus 101 may further selectively include a diffuserbetween the display panel 103 and the sensor panel 107. Also, thedisplay apparatus 101 may prevent the incident first light, to be sensedby the sensor unit 109 of the sensor panel 107, from being blocked bysuch a diffuser posited before or above the sensor panel 107 byarranging the sensor panel 107 to incorporate the diffuser with theaperture 111 of the sensor panel 107, or when one or more of thebacklight units 113 are arranged on the sensor panel 107 instead of theaperture 111, the sensor panel 107 may also include a diffusing layeratop or before each backlight unit 113 to diffuse the light radiated byeach backlight unit 113.

Additionally, though the refresh rate of the display panel may be fixedand frame rates for displayed images would decrease proportionally tothe capture rate, in an embodiment, when the pattern 105 isintentionally not changed upon entering the displaying mode, e.g., evenif there may be available image data for display in the same area as thepattern 105 of the display panel 103, if select apertures 111neighboring a sensor unit(s) 109 corresponding to the pattern 105 arecontrollable to be opaque, or neighboring backlighting units 113incorporated into the sensor panel 107 are controlled to not emit theirrespective second lights, during the displaying mode, an image may becaptured from the specific sensor unit(s) 109 at the exact time an imageis being displayed through all other portions of the display panel 103depending on the ambient light effect of neighboring second lights thatare radiating through other apertures 111 or emitted by otherneighboring backlighting units 113. In such an embodiment, the displaymay operate at its full refresh rate and images may be captured at theirfull capture rate.

However, as the ambient effects may make such same-time capturingimpracticable, alternative embodiments may include the display apparatus101, or the display device 1100 of FIG. 11A, having the respectivedisplay panels 103 specifically engineered to reduce the ambient lighteffect, e.g., for particular areas of the display. As an alternative,the display apparatus 101 or display device 1100 of FIG. 11A may have afunctional mode, for example, that would permit additional neighboringapertures to be made opaque or additional backlighting units 113 to becontrolled to not emit their respective second lights to reduce theambient light effect, in which case the controller could also change avertical or horizontal scanning ratio or aspect ratio of the displayedimages to compensate for the loss in viewable space of the display panel103. Herein, the term ‘simultaneous’ with regards to displaying an imageand capturing an image means that image capturing can be performednearly at the same time as image displaying, potentially with eithermode following the other in a directly subsequent cycle. Herein,regarding a capturing of an image and displaying of an image, the term‘same time’ means that the displaying and displaying occur within thesame cycle. When capturing and displaying of respective images occur inat the same the sampling rate and display rate, e.g., frames per second,may sum up to be greater than the actual refresh rate of the displaypanel 103. Accordingly, in an embodiment, the controller 115 may controla capture rate for capturing images using the sensor layer 107 and adisplay rate for displaying images using the display panel 103, at thesame time to be in total greater than the maximum refresh rate of thedisplay panel 103. Further, the controller 115 may selectively performthe capturing mode and displaying mode simultaneously or at the sametime.

The controller 115 may control the display panel 103 to enable theacquiring or capturing of an object image in the capturing mode and thedisplaying of image data in the displaying mode by switching the mode ofthe display panel 103 to/from the capturing mode or the displaying modebased on a predetermined criterion. As noted above, in an embodiment thepredetermined criterion may merely be whether the previous operationmode was the capturing mode or the displaying mode, and to currentlyoperate in a different mode, e.g., in an alternating manner. However, inone or more embodiments, the percentage of the cycles, such as thepercentage of cycles in the 120 available cycles (120 Hz refresh rate)of an example LCD panel, for each of the capturing mode and thedisplaying mode may not be uniform.

In an embodiment, more cycles may be designated for the capturing mode,or more cycles may be designated for the displaying mode. In anembodiment, the number of cycles a same mode operation is maintained mayalso change during the total 120 available cycles, as the number ofcycles may vary between such same mode operations that the other modeoperates. These variances between operation modes may be controlled bythe controller 115, for example, and may be based on different factors,such as the desired image capturing resolution, the desired imagecapturing frame-rate, the desired image displaying resolution, or thedesired image displaying resolution. These desires may be usercontrolled and stored in a memory of the display device, and/or may bebased on external factors, such at the amount of available light forcapturing image, the viewing environment where displayed image is beingviewed, the data transportation capabilities or settings between thedisplay device and another display device, such as in teleconferencing,where the captured image data would be encoded and transmitted, eitherwired or wirelessly, to the other display device.

Here, in an embodiment, when the other display device, as an exampleremote display device, is also a display apparatus 101, the remotedisplay device may also be respectively performing capturing mode anddisplaying mode operations and respectively equally controlling any ofthe above controls on image capturing and/or displaying. In anembodiment, the capturing mode and displaying mode configurations andoperations of an example local display device may control the localcapturing mode and displaying mode configurations and operations basedon the capturing mode and displaying mode configuration and operationsettings of the remote display device. Accordingly, in one or moreembodiments, encoded image data may be coupled with informationidentifying the settings of the local display device. Here, the remoteor local display devices may also decode this information identifyingthe settings of the respective local or remote device. The informationmay further include indications of why the settings were instituted,such as due limitations in processing or display/capturing capability ofeither device, or transmission channel limitations.

When the display panel 103 is controlled to be in the capturing mode,the controller 115 may adjust a size of one or more of the patternsformed on the display panel 103 based on a distance between the displaypanel 103 and the sensor panel 107, or a size of the sensor panel 107.For example, the controller 115 may be capable of being used indifferently sized display devices, with different resolutions, and/orwith different distances between the respective display panel 103 andsensor panel 107, e.g., in an embodiment of the display apparatus 101where the distance can be modified or when the controller is firstdetermining thee appropriate size or location for the pattern 105 anddetermines the internal dimensions of the display apparatus 101. One ormore embodiments include a system having plural display devices withrespective capturing mode capabilities through respective sensor panels107 and controllers 115, with each of the configurations of the displaydevices not being required to have the same configuration, such as theavailable different arrangements of the pattern 105, sensor units 109,aperture 111, backlight 113, and diffusing elements. Still further, inan embodiment when there are more than two such display devices and morethan two remotely generated images are displayed on the local displaypanel 103, densities of the sensor units 109 in the local sensor panel107 may be greater in the areas corresponding to the displayed images,compared to densities of the sensor units 109 in other portions of thesensor panel 107. With such an arrangement, both remote viewers may beprovided with the feeling that the local viewer is looking directly ateach of them.

The controller 115 may control a power of the backlight unit 113associated with a light emission towards the display panel 103 based onthe mode of the display panel 103 and thereby control the light emittedfrom the backlight unit 113.

For example, when the display panel 103 is in the capturing mode, thecontroller 115 may power off the backlight unit 113 so that the secondlight may not be emitted from the backlight unit 113 towards the displayunit 103, so the second light does not interfere with the sensing of theincident first light. When the display panel 103 is controlled to be inthe displaying mode, the controller 115 may power on the backlight unit113 so that the second light may be emitted from the backlight unit 113towards the display unit 103. Alternatively, in an arrangement where thebacklight unit 113 is below the sensor panel 107, the aperture 111 maybe configured to be controllable to change opacity from opaque totransparent to selectively block the second light, so the back lightwould not have to cycle on and off.

The controller 115 may control a sensor image restoring unit, includedin the controller 115, as only an example, to interpolate image dataacquired by sensing the incident first light using one or more sensorunits 109 of the display panel 103, and to restore image data lost dueto the inclusion of the aperture 111 in the sensor panel 107, comparedto a configuration where the aperture 111 was replaced by another sensorunit 109 that would have sensed the incident first light incident on thesame position of the sensor panel 107. Accordingly, the controller 115may acquire the image data that may more clearly express thecorresponding object. In an embodiment, different senor units 109 orgroups of sensor units 109 may be capturing image information fordifferent purposes. For example, a sensor unit 109 or a group of sensorunits 109 may be capturing image data for transmission to a remoteviewing device, while alternate sensor units 109 or groups of sensorunits 109 may be capturing image data for motion detection or depthdetection. The depth detection may be used for encoding the capturedimage data in a 3D graphics data format, for example, of used forcontrolling a focus or image compensation based on depth. The motiondetection may be used with both on-surface (touching the surface of thedisplay device) motion detection and/or off-surface (not touching thesurface of the display device) motion detection, for detecting gesturesof the user, for example, which may be used to control operations of thedisplay device. In one or more embodiments, the controller 115 controlsthe separate detections, the on/off surface motion detection, and thedepth detection. Depending on computational complexity, some detectionprocesses may be delegated to alternate processing elements of thedisplay apparatus 101 or display device 1100, such as shown in FIG. 11A.

Accordingly, one or more embodiments above enable a user to manipulateoperations of the display apparatus 101 through a multi-touch interfacebased at least on the captured image data by the sensor panel 107,either by directly touching the display or through off-surface gesturesor other detectable imaging features that can be inferred or set as acontrol of the display apparatus 101. Control of the display apparatusis available by more then the image detected gestures, such as based ondetected environments surrounding the display apparatus 101 or featuresdetected in the facial image of the viewer, including for facialrecognition for secure use of the display apparatus 101 or secure accessof remote information. The mood of the viewer may also be detected. Oneor more embodiments include an interface technology that enables anatural interaction even though the user is at a distance from a screen,such as proximate sensing and touching, gesture recognition andlong-distance gesture input, e.g., beyond multiple touches availablethrough on-surface input methods, and detection of a camera level. Suchmulti-touch interface detection, off-surface gesture detection,off-surface image feature detection, facial recognition, emotiondetection, and proximate and long-distance gesture detection and inputinterface are implemented in one or more embodiments based onconventional known techniques, though the capturing of the necessaryimage and depth information, and transmission and reception of anyavailable remote image data and capture configurations, for example, canonly be achieved through the arrangements set forth herein.

In previous approaches, it was not possible to have eye contact whenoperating a display device in a video call mode, so with the previousapproaches users would not take advantage of such video call modes sincethe user's would not feel that they were viewing each other's face dueto the non-matching between the screen position and camera position.Similarly, due to the drawbacks of the previous approaches, users ofsuch devices would not feel as if the user was physically with a counterparty at a very remote location using a large screen such as ImmersiveWall and Smart Window, as when the users moved closer to the screen, thefront image of the user would be lost because of the necessary distantlocation of the camera and the moving user. As another example, previousapproaches were unable to simultaneously perform capturing of the imageof the user while permitting the user to also control the display devicethrough a gesture interface, as the camera for the capturing of theimage of the user are typically positioned at an upper end of thedisplay, which makes gesture detection difficult. Similarly, when cameraplacement was more appropriate for gesture detection, the placement ofthe camera would not be appropriate for image capturing of the user.Even with the Hirsch et al. system described above, when a camera waspositioned behind the display panel, the image resolution was inadequatefor image capturing and the frame rate would not be satisfactory foraccurate gesture detection.

However, these drawbacks are all overcome by one or more of the abovedescribed embodiments.

An operation of the display apparatus may be more fully understood byreference to FIG. 1B, which illustrates the operation of the displaypanel 103 and sensor panel 107 in acquiring image data with respect toan object with a pinhole camera, according to one or more embodiments.Here, the pinhole camera represents at least the pattern 105 of FIG. 1A,for example. Discussions below regarding operations of the pinholecamera, such as calculations being performed, may be performed by thecontroller 115 of FIG. 1A, or alternative processing devices, such as inthe display device 1100 of FIG. 11A.

Referring to FIG. 1B, the pinhole camera corresponds to a camera formingan image on a sensor pane by employing a pinhole instead of using lensesto focus the incident light. Accordingly, the pinhole camera may use, asthe pinhole, pattern 105 in the display panel 103, and may use thesensor panel 107 as the sensor plane. When a plurality of patterns areformed on the display panel 103, an effect of a plurality of cameras maybe provided.

Two types of blurs may occur based on the size of the pinhole, e.g.,pattern 105 of the display panel 103. For example, when an aperturediameter of the pinhole is relatively large, a blur by a geometriceffect may occur. When the aperture diameter of the pinhole isrelatively small, a blur by a diffraction effect may occur.

The blur b_(G) by the geometric effect may be expressed by the belowEquation 1, for example.

$\begin{matrix}{b_{G} = \frac{a\left( {d_{o} + d_{i}} \right)}{d_{o}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In Equation 1, d_(o) denotes a distance between an object and thepinhole, a denotes the aperture diameter, and d_(i) denotes a distancebetween the pinhole and the sensor plane.

With respect to a wavelength λ of light, the blur b_(D) caused by theeffect of optical diffraction may be expressed by the below Equation 2,for example

$\begin{matrix}{b_{D} = \frac{2.44\lambda \; d_{i}}{a}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

In Equation 2, d_(i) denotes the distance between the pinhole and thesensor plane.

A size b of a final blur corresponds to a sum of the two effects andthus, may be expressed by the below Equation 3, for example.

$\begin{matrix}{b = {{b_{G} + b_{D}} = {\frac{a\left( {d_{o} + d_{i}} \right)}{d_{o}} + \frac{2.44\lambda \; d_{i}}{a}}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

When the aperture diameter and the distance between the pinhole and thesensor plane are known, the pinhole camera may calculate the size b ofthe blur and calculate the depth of the object, or distance between theobject and the pinhole, according to the below Equation 4, for example

$\begin{matrix}{d_{o} = \frac{a^{2}d_{i}}{{ab} - {2.44\lambda \; d_{i}} - a^{2}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

Here, using Equation 4, the distance between the object and a displaypanel, or the depth of the object, may be calculated based upon the twoblurs. Briefly, it is noted that this is different than previousapproaches, such as in the above referenced Hirsch et al. article, wherethe authors use Equation 3 (as Equation (1) in Hirsch et al.) for adifferent purpose to achieve a different answer; the authors usedinformation from the two blurs to determine the angular resolution oftheir system to control the number of orthographic views, rather thanfor obtaining depth information.

The pinhole camera may calculate an inverse impulse response based onthe distance d_(o) between the object and the pinhole. The pinholecamera may further restore distance-based image data by performingde-convolution using the inverse impulse response with respect to theblurred image data acquired by the sensor plane, as shown in the belowEquation 5, for example.

|(x, y)={tilde under (|)}′*H ⁻¹(x, y)  Equation 5:

In Equation 5, | (x, y) denotes the restored image data, {tilde under(|)}′ (x, y) denotes the blurred image data, and H⁻¹ (x, y) denotes theinverse pulse response.

Accordingly, the pinhole camera may acquire more accurate image databased on determined distance between the object to the pinhole. Usingthe display apparatus 101 of FIG. 1A as an example, the controller 115may accordingly determine the distance between the object and thepattern 105 and then restore the image captured by corresponding one ormore sensor units 109 based on that determined distance.

FIGS. 2A and 2B illustrate a sensor panel 201, such as the sensor panel107 of FIG. 1A, of a display apparatus, according to one or moreembodiments.

Referring to FIG. 2A, the sensor panel 201 may include a sensor unit 203to sense incident light and an aperture 205 to pass light radiating frombelow the sensor panel 201. The sensor panel 201 may be configured suchthat the sensor unit 203 and the aperture 205 are arranged in a gridpattern, or may be configured such that the sensor unit 203 and theaperture 205 are arranged in a repeating pattern. Through this, in anembodiment, the sensor units 203 and the apertures 205 may be uniformlydistributed.

Accordingly, in this embodiment, the sensor panel 201 may uniformlyacquire an incident first light, e.g., reflected from an object, byhaving the sensor units 203 and the apertures 205 configured to bedistributed, and may uniformly transfer, to an upper display panel, asecond light radiating from a backlight unit below the sensor panel 201.

When a pattern included in a pattern of the display panel in thecapturing mode is formed based on a pixel unit of the display panel,including R, G, and B pixel units, the sensor panel 201 may acquireimage data corresponding to R, G, and B by sensing the light passedthrough a pattern 211. Also, when the pattern included in the pattern ofthe display panel is formed based on a sub-pixel unit, a pattern 213formed as an R sub-pixel, a pattern 215 formed as a G sub-pixel, and apattern 217 formed as a B sub-pixel may be sequentially formed.Accordingly, the sensor panel 201 may acquire image data correspondingto R, G, and B by respectively sensing the light passing through thepatterns 213, 215, and 217.

Configurations of the sensor pattern 201 are not limited to those shownin FIG. 2A, and thus, configurations of the sensing units to sense theincident light and apertures to pass the radiating light may vary asshown in the different configurations shown in the left, center, andright sensor panel illustrations of FIG. 2B.

FIGS. 3A and 3B illustrate sensor panels 301 and 307, such as the sensorpanel 107 of FIG. 1A, of a display apparatus, according to one or moreembodiments.

Referring to FIG. 3A, the sensor panel 301 may be configured to includea sensor unit with a color filter to sense incident light that haspassed through a pattern of a display panel, such as the pattern 105 ofdisplay panel 103 in FIG. 1, and to acquire respective image datacorresponding to the color of the color filter. The sensor panel 301 mayinclude a first sensor unit and a second sensor unit, for example. Eachof the first sensor unit and the second sensor unit may also includerespective differently colored unit color filters. Additionally, each ofthe first sensor unit and the second sensor unit may include a pluralityof respective differently colored color filters. The first sensor unitsand the second sensor units may be organized according to any Bayerpattern, for example, and different first sensor units and second sensorunits may be filter one or more different colors through different Bayerpattern.

As one example, as shown in the sensor panel 301 of FIG. 3A, the sensorpanel 301 may be configured to have a first sensor unit 303 as red colorunit color filter and to have a second sensor unit 305 as a blue colorunit color filter. As another example, as shown in the sensor panel 307of FIG. 3A, the sensor panel 307 may be configured to have each of thefirst sensor unit 309 and the second sensor unit 311 include a pluralityof RGB color filters.

The configuration of the sensor panel is not limited to those shown insensor panels 301 and 307 of FIG. 3A, and thus, the sensor panel may beconfigured to have various types of color filters as shown the differentconfigurations shown in the upper left, upper right, lower left andlower right sensor panel illustrations in FIG. 3B.

FIGS. 4A-4B illustrates a sensor panel 401, such as the sensor panel107, of a display apparatus, according to one or more embodiments.

Referring to FIG. 4A, the sensor panel 401 may include a sensor unit 403to sense incident light and a backlight unit 405 to radiate lighttowards a display panel 407. In one or more embodiments, the sensorpanel 401 may be configured such that the sensor unit 403 and thebacklight unit 405 are configured in a grid pattern, or the sensor unit403 and the backlight unit 405 may be configured in a repeating pattern,and thus, in one or more embodiments, the sensor units 403 and thebacklight units 405 may be uniformly distributed.

Accordingly, in such an embodiment, the sensor panel 401 may beconfigured to uniformly acquire incident first light, e.g., reflectedfrom an object, and to uniformly radiate light. Here, the sensor units403 and the backlight units 405 may be arranged so they are evenlydistributed. Accordingly, the sensor panel 401 may uniformly acquireincident light from sensor units 403 and uniformly radiate light fromthe backlight units 405 to the display panel 407.

Since the sensor panel 401 includes the backlight unit 405, the lightradiated from the backlight unit 405 to the display panel 407 may bedirectly emitted towards the display panel 407, instead of being emittedtowards the display panel 407 through the sensor panel 401, e.g., asshown in FIGS. 2A and 2B. FIG. 4B also illustrates the sensor panel 401and the display 407, but demonstrates where there are two pinholes ortwo patterns 105 in the display panel 407, which creates a capturedstereo image by respective sensor units 403.

Accordingly, the sensor panel 401 and the backlight unit 405 may beintegrally formed, and thus, it is possible to reduce the internalvolume of the display apparatus, by removing the previously requiredbacklight unit disposed below the sensor panel 401, thereby producinglight with a slim display apparatus 409. Additionally, with thisconfiguration there is an increase in efficiency of light emission overthe arrangement of FIGS. 2A and 2B, for example, as light from a belowbacklight is not being limited to transmission through only theapertures 111.

FIGS. 5A-5B illustrate a sensor panel 501, such as the sensor panel 107of FIG. 1A, in a display apparatus, according to one or moreembodiments.

Referring to FIG. 5A, the sensor panel 501 may include a sensor unit 503to sense incident light and a diffuser 505 to diffuse light radiatingfrom a backlight unit 509. The sensor panel 501 may be configured tohave the sensor unit 503 and the diffuser 505 arranged in a gridpattern, or may be configure to have the sensor unit 503 and thediffuser 505 arranged in a repeating pattern, as only examples.Accordingly, in an embodiment, the sensor units 503 and the diffusers505 may be uniformly distributed.

In such an embodiment, the sensor panel 501 may uniformly acquireincident first light, e.g., reflected from an object, by having thesensor units 503 and the diffusers 505 distributed evenly, and thus, mayuniformly radiate a second light emitted from the backlight unit 509 tothe display panel 507.

According to an embodiment, the display apparatus 511 using the sensorpanel 501 including the diffuser 505 may directly sense incident light,passed through the display panel 507, using the sensor panel 501, andthereby acquire accurate image data with respect to an object, forexample. FIG. 5B also illustrates the sensor panel 501 and the display507, but demonstrates where there are two pinholes or two patterns 105in the display panel 507, which creates a captured stereo image byrespective sensor units 503.

FIGS. 6A through 6C, and FIG. 7 illustrate plural available patternsthat may be formed in a display panel 601, such as the display panel 103of FIG. 1A, in a display apparatus, according to one or moreembodiments.

Referring to FIG. 6A, the display panel 601 may be controlled to form apattern 603 by controlling a first pixel to be opaque and by controllinga second pixel, different from the first pixel, to be transparent, andmay form a pattern to be included in the pattern 603 so that a light,e.g., reflected from an object, may pass through the pattern 603.

The display panel 601 may be controlled to form the pattern 603 toinclude a circular hole pattern. However, as this is only an example,the display panel 601 may be controlled to form a pattern 605, e.g.,including a polygonal hole, and a pattern 607 including a MURA image.

As shown in FIG. 6B, the display panel 601 may be controlled to form apattern of 11×11 pixels (or sub-pixels), in illustration (i), a patternof 17×17 pixels (or sub-pixels), in illustration (ii), and/or a patternof 23×23 pixels (or sub-pixels), in illustration (iii). As shown in FIG.6C, the display panel 601 may equally be controlled to form a MURApattern of 11×11 pixels (or sub-pixels), in illustration (i), a MURApattern of 17×17 pixels (or sub-pixels), in illustration (ii), and/or aMURA pattern of 23×23 pixels (or sub-pixels), in illustration (iii). Thedisplay panel 601 may be controlled to form both a pattern and a MURApattern, with varying pixel or sub-pixel sizes. The illustration (i) ofFIG. 6B may also represent a pattern using a single pixel or singlesub-pixel.

As shown in FIG. 7, the display panel 601 may form a plurality ofpatterns and may pass incident light, e.g., reflected from an object,using a plurality of patterns. The left illustration of FIG. 7demonstrates the above-referenced stereo image, which may be useful forimage capturing of a viewer, while the center and right illustrations ofFIG. 7 demonstrate patterns that may be useful for gesture capturing ororthographic capturing of a view or object with greaterthree-dimensional data, for example. In one or more embodiments, evenwhen plural patterns are used, the controller 115 of FIG. 1A, forexample, may resolve the images both for depth and use some or all ofthe collected image information for generating a single image or one ormore less images than patterns.

FIG. 8 through FIG. 10 illustrate patterns that may be formed by adisplay panel, such as by the display panel 103 of FIG. 1A, included ina display apparatus, according to one or more embodiments.

Here, the display panel may be controlled to form a plurality ofpatterns, and may adjust the number of patterns and/or the positions ofeach pattern by changing at least one of a number of opaque firstpixels, a number of transparent second pixels, a position of each opaquefirst pixel, and a position of each transparent second pixel, forexample.

Referring to FIG. 8, the display panel may horizontally sequence theplurality of patterns by sequentially forming the plurality of patternsin a first column 801, a second column 803, a third column 805, and afourth column 807, at predetermined intervals, e.g., in different cyclesof the available 120 cycles of an example 120 Hz refresh rate of an LCDscreen.

Referring to FIG. 9, the display panel may be controlled to verticallysequence the plurality of patterns by sequentially forming the pluralityof patterns in a first line 901, in a second line 903, and a third line905, at predetermined intervals, e.g., in different cycles.

Referring to FIG. 10, the display panel may be controlled to alternatelyform the plurality of patterns, e.g., by sequentially changing patternpositions according to an interleaving scheme.

FIG. 11A through 11C illustrate a display device, system, and method,according to one or more embodiments.

Referring to FIG. 11A, the display device 1100 includes a display anduser interface 1101, a controller 1115, a multimedia decoder 1120, amultimedia encoder 1130, a central processing unit (CPU), a memory 1150,transmitter/receiver 1160, speaker 1170, and microphone 1180, forexample. The display and user interface 1101 may operate similarly toany of the above described display panels, sensor panels, sensor units,apertures, backlight units, and diffusing arrangements, with thecontroller 1115 operating as the controller 115 of FIG. 1A, for example.The captured image information may be analyzed by the controller 1115 orthe necessary processing may be handed off to the CPU, such as wheremore extensive processing is required in gesture detection, facialdetection, image resolution or convolution, and the like. The displayand user interface 1101 may include user inputs through on/off-screenmotion detection or gesture detection, as only example. The encoder 1130may encode the captured image data, including performing any 3D graphicsdata encoding based upon the determined depth of one or more objects bythe controller 1115, for example. The encoder may also encode capturedaudio data with the encoded image data, e.g., captured through themicrophone 1180, according to any conventional video/audio encodingschemes, such as any MPEG standard, as only an example.

The encoder may also encode with or separately the encoded video andaudio information and control information, e.g., the above discussed theenvironment or settings for the capturing and/or displaying modes andcorresponding set configurations and arrangements, as only example, usedin the image capturing which may be considered by a remote displaydevice, such as in the system of FIG. 11B having first and seconddisplay devices 1100-1 and 1100-2.

The transmitter/receiver 1160 may then transmit the encoded data to theremote display device, such as the second display device 1000-2 of FIG.11B. The transmitter/receiver 1160 may also receive similarly encodedinformation from the remote display device and forward the same to thedecoder 1120, which decodes the video/audio data and any control datafrom the remote display device. The decoded video/audio information isthen output through the display and any decoded control information isreviewed by the controller to determine whether to change the currentimage capturing settings of capturing mode of the display device 1100.

Referring to FIG. 11B, the system 1110 includes the display device 1100and the remote display device of FIG. 11A, as the first display device1100-1 and second display device 1100-2, in one or more embodiments,each being a display device corresponding to the display device 1100 ofFIG. 11A, e.g., including a display and user interface 1101 and encoderand decoder elements 1125. The network 1190 may be any communicationpath available between the first display device 1000-1 and the seconddisplay device 1100-2, such as an Internet Protocol based network orwireless protocol, or a combination of the same. Both first and seconddisplay devices 1100-1 and 1100-2 do not need to be a display devicecorresponding to the display device 1100 of FIG. 11A, and accordingly,it may not be necessary to communicate between display devices how anytransmitted image data was generated.

Referring to FIG. 11C, an alternative view of the display device 1100 ofFIG. 11A is shown, with the viewer using a video call capability of thedisplay device 1100. Though illustrated in FIG. 11C as a mobile phone orsmart phone, the display devices 1100 of FIGS. 11A and 11B, as the firstor second display devices 1101-1 and 1101-2 in the system 1110, may beany same or different type of display device in the system 1110,including televisions, a personal computer and display as a personalcomputer system, tablet computer devices, mobile phones, PDAs,teleconferencing devices, set-top boxes, etc. The system 1110 of FIG.11B is intended to include any of these devices as either of the firstor second display devices 1101-1 and 1101-2, without requiring the firstor second display devices 1101-1 and 1101-2 to be the same type ofdisplay device.

In one or more embodiments, any apparatus, system, and unit descriptionsherein include one or more hardware devices and/or hardware processingelements/devices, e.g., controllers incorporated into the displayingsystem or with hardware elements. For example, one or more embodimentsinclude a controlling method and/or a controller device controlling thedisplay panel to generate the pinhole apertures during the capturingmode and controlling the display panel to display an image during thedisplaying mode, distinctly from a controller that may control thesensor panel, or diffusing aspects. Each of these controllers mayinclude such hardware processing elements. Additionally, one or moreembodiments may include a configuration similar to that of FIG. 11A,including one or more processing elements in the controller, CPU,display, encoder, and/or decoder hardware portions of the mobile device.Accordingly, in one or more embodiments, any described apparatus,system, and unit may further include one or more desirable memories, andany desired hardware input/output transmission devices, as onlyexamples. Further, the term apparatus should be considered synonymouswith elements of a physical system, not limited to a device, i.e., asingle device at a single location, or enclosure, or limited to alldescribed elements being embodied in single respective element/device orenclosures in all embodiments, but rather, depending on embodiment, isopen to being embodied together or separately in differing devices orenclosures and/or differing locations through differing hardwareelements.

In addition to the above described embodiments, embodiments can also beimplemented through computer readable code/instructions in/on anon-transitory medium, e.g., a computer readable medium, to control atleast one processing element/device, such as a processor, computingdevice, computer, or computer system with peripherals, to implement anyabove described embodiment or aspect of any embodiment. The medium cancorrespond to any defined, measurable, and tangible structure permittingthe storing and/or transmission of the computer readable code.Additionally, one or more embodiments include the at least oneprocessing element or device.

The media may also include, e.g., in combination with the computerreadable code, data files, data structures, and the like. One or moreembodiments of computer-readable media include magnetic media such ashard disks, floppy disks, and magnetic tape; optical media such as CDROM disks and DVDs; magneto-optical media such as optical disks; andhardware devices that are specially configured to store and/or performprogram instructions, such as read-only memory (ROM), random accessmemory (RAM), flash memory, and the at least one processing device,respectively. Computer readable code may include both machine code, suchas produced by a compiler, and files containing higher level code thatmay be executed by the computer using an interpreter, for example. Themedia may also be any defined, measurable, and tangible elements of oneor more distributed networks, so that the computer readable code isstored and/or executed in a distributed fashion. In one or moreembodiments, such distributed networks do not require the computerreadable code to be stored at a same location, e.g., the computerreadable code or portions of the same may be stored remotely, eitherstored remotely at a single location, potentially on a single medium, orstored in a distributed manner, such as in a cloud based manner. Stillfurther, as noted and only as an example, the processing element couldinclude a processor or a computer processor, and processing elements maybe distributed and/or included in a single device. There may be morethan one processing element and/or processing elements with pluraldistinct processing elements, e.g., a processor with plural cores, inwhich case one or more embodiments would include hardware and/or codingto enable single or plural core synchronous or asynchronous operation.

The computer-readable media may also be embodied in at least oneapplication specific integrated circuit (ASIC) or Field ProgrammableGate Array (FPGA), as only examples, which execute (processes like aprocessor) program instructions.

While aspects of the present invention has been particularly shown anddescribed with reference to differing embodiments thereof, it should beunderstood that these embodiments should be considered in a descriptivesense only and not for purposes of limitation. Descriptions of featuresor aspects within each embodiment should typically be considered asavailable for other similar features or aspects in the remainingembodiments. Suitable results may equally be achieved if the describedtechniques are performed in a different order and/or if components in adescribed system, architecture, device, or circuit are combined in adifferent manner and/or replaced or supplemented by other components ortheir equivalents.

Thus, although a few embodiments have been shown and described, withadditional embodiments being equally available, it would be appreciatedby those skilled in the art that changes may be made in theseembodiments without departing from the principles and spirit of theinvention, the scope of which is defined in the claims and theirequivalents.

1. A display apparatus, comprising: a sensor panel configured to sense afirst incident light, incident to the sensor panel subsequent to passingthrough a display, and to at least partially block transmission of lightthrough the sensor panel; and a controller to selectively control thedisplay to be configured to pass the first incident light toward thesensor panel and to control the display to be configured to pass lightout of the display to display an image, wherein the sensor panelcomprises plural distinct portions configured to each forward light fromthe sensor panel towards the display.
 2. The display apparatus of claim1, wherein the controller selectively controls the display to only passthe first incident light toward the sensor panel in a first cycle, ofplural fixed time length cycles, and controls the display to only passthe light out of the display to display the image in a second cycle, ofthe plural fixed time length cycles.
 3. The display apparatus of claim2, wherein the fixed time is a time it takes for the display to performone refresh.
 4. The display apparatus of claim 1, wherein, when thecontroller selectively controls plural images to be captured from pluralsensings of respective first incident lights, representing a capturerate, from a pattern of the display controlled by the controller to notchange upon the controller controlling the display to display the image,the controller is configured to control the capture rate plus a displayrate, for displaying plural images, to be greater than a maximum refreshrate of the display.
 5. The display apparatus of claim 1, wherein thecontroller selectively controls opacities of pixels and/or sub-pixels ofthe display to produce patterns on the display for displaying the imageon the display, in a second mode, and to form at least one pattern onthe display controlled to be transparent and an area of the displaysurrounding the at least one pattern controlled to be opaque, to capturean image by at least one sensor unit of the sensor, in a first mode,wherein the display is controlled by the controller differently in thefirst mode and the second mode.
 6. The display apparatus of claim 5,wherein the controller determines a depth of an object represented inthe first incident light based upon at least one detected size of a blurof the at least one pattern detected by the sensor panel.
 7. The displayapparatus of claim 6, wherein the controller determines plural depthmeasurements of the object represented in plural first incident lightseach sensed by the sensor panel based respectively upon at least onedetected size of a blur pattern for each sensed incident light, andgenerates 3D information of the object.
 8. The display apparatus ofclaim 7, wherein the controller controls a monitoring of movement by theobject over plural captured images and respectively generated plural 3Dinformation and determines whether the monitored movement and generatedplural 3D information indicates that a gesture, of plural definedgestures, is being made by the movement of the object.
 9. The displayapparatus of claim 6, wherein the controller restores the captured imageby calculating a distance-based impulse response based on the determineddepth and by performing devolution of the captured image using thedistance-based impulse response to compensate for refraction and/orgeometric effects caused by the at least one pattern of the display. 10.The display apparatus of claim 1, further comprising: a display panel,as the display; and a backlight unit to emit a second light towards thesensor panel, wherein the controller selectively controls the displaypanel for passing the first incident light toward the sensor panel, in afirst mode, and controls the display panel for passing the second lightthrough the display panel to display the image, in a second mode. 11.The display apparatus of claim 10, wherein the plural distinct portionseach are configured to always forward the second light from thebacklight unit towards the display unit.
 12. The display apparatus ofclaim 10, wherein the controller controls the backlight unit to emitlight in the second mode and to not emit light during the first mode.13. The display apparatus of claim 10, wherein the display device is oneof a television, a personal computer system, a tablet computer device, amobile phone, a PDA, teleconferencing devices or system, or set-top box,with each including at least one processor and memory, an encoder toencode an image captured by the sensing by the sensor panel in the firstmode, a decoder to at least decode image data received from an externalsource, and with the controller forwarding decoded image data from thedecoder to the display panel and controlling the display panel todisplay the decoded image data in the second mode.
 14. The displayapparatus of claim 13, wherein the controller selectively controlsopacities of pixels and/or sub-pixels of the display panel to producepatterns on the display panel for displaying the image on the displaypanel, in a second mode, and to form at least one pattern on the displaypanel controlled to include at least one transparent pixel or sub-pixeland an area of the display panel surrounding the at least one patterncontrolled to be opaque, to capture an image by at least one sensor unitof the sensor panel, in a first mode, wherein the display panel iscontrolled by the controller differently in the first mode and thesecond mode.
 15. The display apparatus of claim 13, wherein thecontroller controls a monitoring of movement of an object, representedin the first incident light, over plural captured images andrespectively generated plural 3D information for the object anddetermines whether the monitored movement and generated plural 3Dinformation indicates that a gesture, of plural defined gestures, isbeing made by the movement of the object.
 16. The display apparatus ofclaim 13 wherein the controller restores the captured image bycalculating a distance-based impulse response based on a determineddepth of an object, represented in the first incident light, and byperforming devolution of the captured image using the distance-basedimpulse response to compensate for refraction and/or geometric effectscaused by a pattern of the display panel from where the first incidentlight passed through the display panel, wherein the determined depth ofthe object is calculated by the controller based upon at least onedetected size of a blur of the first incident light, as incident on thesensor panel.
 17. The display apparatus of claim 1, wherein, when thecontroller selectively controls plural images to be captured from pluralsensings of respective first incident lights, representing a capturerate, from a pattern of the display controlled by the controller to notchange upon the controller controlling the display to display the image,the controller is configured to control the capture rate plus a displayrate, for displaying plural images, to be greater than a maximum refreshrate of the display.
 18. The display apparatus of claim 1, furthercomprising: a display panel, as the display; and plural backlight unitsrespectively being the plural distinct portions of the sensor panel,each to respectively emit a second light towards the display panel,wherein the controller selectively controls the display panel to passthe first incident light toward the sensor panel in a first mode andcontrols the display panel to pass each of the second lights through thedisplay panel in a second mode.
 19. The display apparatus of claim 18,wherein the controller controls each of the plural backlight units toemit light in the second mode and to not emit light during the firstmode.
 20. The display apparatus of claim 18, wherein the display deviceis one of a television, a personal computer system, a tablet computerdevice, a mobile phone, a PDA, teleconferencing devices or system, orset-top box, with each including at least one processor and memory, anencoder to encode an image captured by the sensing by the sensor panelin the first mode, a decoder to at least decode image data received froman external source, and with the controller forwarding decoded imagedata from the decoder to the display panel and controlling the displaypanel to display the decoded image data in the second mode.
 21. Thedisplay apparatus of claim 20, wherein the controller selectivelycontrols opacities of pixels and/or sub-pixels of the display panel toproduce patterns on the display panel for displaying the image on thedisplay panel, in a second mode, and to form at least one pattern on thedisplay panel controlled to include at least one transparent pixel orsub-pixel and an area of the display panel surrounding the at least onepattern controlled to be opaque, to capture an image by at least onesensor unit of the sensor panel, in a first mode, wherein the displaypanel is controlled by the controller differently in the first mode andthe second mode.
 22. The display apparatus of claim 20, wherein thecontroller controls a monitoring of movement of an object, representedin the first incident light, over plural captured images andrespectively generated plural 3D information for the object anddetermines whether the monitored movement and generated plural 3Dinformation indicates that a gesture, of plural defined gestures, isbeing made by the movement of the object.
 23. The display apparatus ofclaim 20, wherein the controller restores the captured image bycalculating a distance-based impulse response based on a determineddepth of an object, represented in the first incident light, and byperforming devolution of the captured image using the distance-basedimpulse response to compensate for refraction and/or geometric effectscaused by a pattern of the display panel from where the first incidentlight passed through the display panel, wherein the determined depth ofthe object is calculated by the controller based upon at least onedetected size of a blur of the first incident light, as incident on thesensor panel.
 24. The display apparatus of claim 1, further comprising abacklight unit to generate the light to be forwarded by the pluraldistinct portions of the sensor panel, wherein the sensor panelcomprises: plural sensor units to sense the first incident light, in afirst mode; and plural apertures, respectively as the plural distinctportions, to pass the second light emitted from the backlight unittowards the display in first and second modes, with the controllercontrolling the display to display an image in the second mode, whereinthe plural sensor units and plural apertures are arranged in a gridpattern across the sensor panel.
 25. The display apparatus of claim 24,wherein—the apertures respectively include a diffusers to diffuse thesecond light.
 26. The display apparatus of claim 1, wherein: the sensorpanel comprises a first sensor unit and a second sensor unit to sensethe incident first light, and each of the first sensor unit and thesecond sensor unit comprise a unit color filter having a different colorfilter.
 27. The display apparatus of claim 26, wherein plural sensorunits of the sensor panel are collectively organized in different Bayerpatterns.
 28. The display apparatus of claim 1, wherein the sensor panelcomprises a first sensor unit and a second sensor unit to sense theincident first light, and each of the first sensor unit and the secondsensor unit comprise a plurality of color filters respectively filteringdifferent colors.
 29. The display apparatus of claim 1, wherein thecontroller further comprises a sensor image restoring unit tointerpolate image data acquired by the sensing of the first incidentlight and to restore image data that is not captured by any sensing ofthe sensor panel at the plural distinct portions of the sensor panel.30. The display apparatus of claim 1, further comprising a displaypanel, as the display, wherein, in a first mode, the controller controlsthe display panel to selectively form at least one pattern to pass thefirst incident light through the display panel to the sensor panel byselectively controlling respective opacities of pixels or sub-pixels ofthe display panel to be different, controlled differently than when thecontroller controls the display panel to display the image, in a secondmode.
 31. The display apparatus of claim 30, wherein the controllercontrols, at a current time in the first mode, the display panel to haveat least one pattern different than a pattern formed at a previous timewhen the display apparatus was in the first mode, by changing at leastone of a number of first pixels of the display panel that areselectively controlled to be opaque during the first mode, a number ofsecond pixels of the display panel that are selectively controlled to betransparent during the first mode, positions of one or more pixels ofthe display panel that are controlled to be opaque during the firstmode, and positions of one or more pixels of the display panel that arecontrolled to be transparent pixels during the first mode, than in theprevious time.
 32. The display apparatus of claim 30, wherein thecontroller selectively controls the display panel to form, as the atleast one pattern, at least one of a circular pattern, a polygonalpattern, and a modified uniformly redundant array (MURA) by controllingplural first pixels or plural first sub-pixels to be opaque and pluralsecond pixels or plural second sub-pixels to be transparent within apredetermined area of the display panel.
 33. The display apparatus ofclaim 32, wherein the predetermined area is one of 11×11 pixels, 11×11sub-pixels, 17×17 pixels, 17×17 sub-pixels, 23×23 pixels, and 23×23sub-pixels.
 34. The display apparatus of claim 1, wherein the sensorpanel is constructed of a substrate material that does not transmitlight.
 35. A display apparatus, comprising: a display panel; a sensorpanel including plural sensor units configured to sense incident lightfrom the display panel and plural distinct portions configured to eachforward light from the sensor panel towards the display panel; and acontroller to selectively control opacities of pixels and/or sub-pixelsof the display panel to produce patterns on the display panel fordisplaying an image on the display panel, in a second mode, and to format least one pattern on the display panel controlled to be transparentand an area of the display panel surrounding the at least one patterncontrolled to be opaque, to capture an image by the at least one sensorunit, in a first mode, wherein the plural distinct portions arerespectively plural apertures or plural backlight units, with one ormore of the plural apertures including a diffuser element, and with eachof the plural backlight units respectively generating and directinglight toward the display panel.
 36. A display method, comprising:selectively configuring a display to form a pattern with at least onetransparent pixel or sub-pixel to pass a first incident light throughthe display and toward a sensor panel, behind the display, and to forman opaque area that at least surrounds the pattern, when a current modeis a first mode, and configuring the display to pass light from abacklight, behind the display, through the display, when the currentmode is a second mode; sensing the first incident light, having passedthrough the display, upon incidence to the sensor panel that at leastpartially blocks transmission of light, in a first mode; displaying animage on the display by passing light in a direction from the sensorlayer through the display; and repeatedly controlling the current modeto only one of the first mode and the second mode, including at least achanging of the current mode plural times.
 37. The method of claim 35,further comprising controlling, at a current time in the first mode, thedisplay to have at least one pattern different than a pattern formed ata previous time when the current mode was the first mode, by changing atleast one of a number of first pixels of the display that areselectively controlled to be opaque during the first mode, a number ofsecond pixels of the display that are selectively controlled to betransparent during the first mode, positions of one or more pixels ofthe display that are controlled to be opaque during the first mode, andpositions of one or more pixels of the display that are controlled to betransparent pixels during the first mode, than in the previous time. 38.The method of claim 35, further comprising controlling, at a currenttime in the first mode, the display to have at least one patterndifferent than a pattern formed at a previous time when the current modewas the first mode, by changing at least one of a number of firstsub-pixels of the display that are selectively controlled to be opaqueduring the first mode, a number of second sub-pixels of the display thatare selectively controlled to be transparent during the first mode,positions of one or more sub-pixels of the display that are controlledto be opaque during the first mode, and positions of one or moresub-pixels of the display that are controlled to be transparent pixelsduring the first mode, than in the previous time.
 39. The method ofclaim 35, further comprising selectively controlling a combination of acapture rate, for capturing images from the sensor panel, and a displayrate, for displaying plural images, to be greater than a maximum refreshrate of the display.
 40. The method of claim 35, further comprisingselectively controlling the display to only pass the first incidentlight toward the sensor panel in a first cycle, of plural fixed timelength cycles, and controlling the display to only pass the light out ofthe display to display the image in a second cycle, of the plural fixedtime length cycles, wherein the fixed time is a time it takes for thedisplay to perform one refresh.
 41. The method of claim 35, furthercomprising selectively controlling opacities of pixels and/or sub-pixelsof the display to produce patterns on the display for displaying theimage on the display, in a second mode, and to form at least one patternon the display controlled to be transparent and an area of the displaysurrounding the at least one pattern controlled to be opaque, to capturean image by at least one sensor unit of the sensor, in a first mode,wherein the display is controlled differently in the first mode and thesecond mode.
 42. The method of claim 40, further comprising determininga depth of an object represented in the first incident light based uponat least one detected size of a blur of the at least one patterndetected by the sensor panel.
 43. The method of claim 41, wherein thedetermining further comprises determining plural depth measurements ofthe object represented in plural first incident lights each sensed bythe sensor panel based respectively upon at least one detected size of ablur pattern for each sensed incident light, and generating 3Dinformation of the object.
 44. The method of claim 42, furthercomprising monitoring a movement by the object over plural capturedimages and respectively generated plural 3D information and determiningwhether the monitored movement and generated plural 3D informationindicates that a gesture, of plural defined gestures, is being made bythe movement of the object, and controlling an operation of a devicebased on the determined gesture.
 45. The method of claim 41, furthercomprising restoring the captured image by calculating a distance-basedimpulse response based on the determined depth, performing devolution ofthe captured image using the distance-based impulse response tocompensate for refraction and/or geometric effects caused by the atleast one pattern of the display, and encoding the restored capturedimage.
 46. A display method, comprising: selectively configuring adisplay to form a pattern with at least one transparent pixel orsub-pixel to pass a first incident light through the display and towarda sensor panel, behind the display, and to form an opaque area that atleast surrounds the pattern, when a current mode is a first mode, andconfiguring the display to pass light from plural backlights of thesensor panel through the display, when the current mode is a secondmode; sensing the first incident light, having passed through thedisplay, upon incidence to the sensor panel that blocks transmission oflight, in a first mode; displaying an image on the display by passingrespectively generated light from each backlight from the sensor layerthrough the display; and repeatedly controlling the current mode to beone of the first mode and the second mode, including at least a changingof the current mode plural times.